modified xml file hip range because original hopper robot is bugged - its hip range allows self intersection, getting the robot stuck in a certain position, with small probability on initialization:

Is the ppo_rgb.py code updated? Seem to run into a few errors due to evaluating 100 steps instead of the max 600.

Is the ppo_rgb.py code updated? Seem to run into a few errors due to evaluating 100 steps instead of the max 600.

yes, I added ppo_rgb 1 line change to get it to work in new commit - there was no check that success exists in final_info dict in ppo_rgb.py when tensorboard writing, while there is a check in ppo.py

for i in {1..3}; do python ppo_rgb.py --exp_name="pixels_hopseed${i}" --env_id="MS-HopperHop-v1" --num_envs=256 --update_epochs=8 --num_minibatches=8 --total_timesteps=25_000_000 --num-steps=100 --num_eval_steps=600 --gamma=0.995 --seed=${i}; done

performance is worse without the pressure sensor information

It is hard to tell if it's "worse" since this is a different return scale now? Also i tried running again but the performance after 5M steps looks like this:

Looks like it learns to flip forward to go fast now, instead of the original behavior.

My script:

python ppo_rgb.py --exp_name="rgbhopseed1" --env_id="MS-HopperHop-v1" --num_envs=512 --update_epochs=8 --num_minibatches=32 --total_timesteps=20_000_000 --eval_freq=15 --num_eval_steps=600 --num_steps=100 --gamma=0.99 --seed=1 --no-include-state

(merge main for this to work)
This seems to perform better than the script you ran with these hyperparams. It does show some sign that it might learn the "right" behavior (it could also be that our hopper setup affords more optimal flipping than mujocos)

Maybe behavior will be removed if the agent does not have so much power behind its actions. I can try and experiment as well.

This has been my experience with the damping, stiffness, and delta being very important for desired behvior. If the damping especially is too low, then the hopper is unable to upright itself, if it's too high, then it can instead flip to its feet too easily.

can you somehow revert to the settings that got the first video that was working? or was it only working from state inputs

it was only working from state inputs, these are the same controller settings that produced the first videos

Arguably though with the flipping it seems like it achieves the same distance if not further than the standard "behavior". I would then say this is more of a result of the reward function. Since its not possible to get the same exact simulation as mujoco, maybe we can just penalize flipping (just penalize the x/y axis rotation velocity should be sufficient, whichever is the axis pointing to the camera). It's fine if our reward function is slightly different to mujocos.

I find many other repos tackling locomotion like isaac lab / orbit will often add reward terms not meant to make the robot run faster, but to have a smoother gait, which is what we will be doing here essentially.

okay, that sounds good, I will try some experiments with discouraging flipping reward

Maybe behavior will be removed if the agent does not have so much power behind its actions. I can try and experiment as well.

No need to try to experiment, I have found that y axis rotation bounds for reward works well, will run tests and report here later tonight

edit: update - It worked well for a quick check on state obs - to ensure no flipping with large deltas. However, I've tried many, many different things, and unfortunately, any additional reward doesn't fix the problem - in fact, I believe the flipping was originally a local min solution, where hopping would result in greater reward, since the flipping doesn't always activate the standing reward portion of the hopping reward

currently testing out non delta joint pos

Maybe behavior will be removed if the agent does not have so much power behind its actions. I can try and experiment as well.

No need to try to experiment, I have found that y axis rotation bounds for reward works well, will run tests and report here later tonight

edit: update - It worked well for a quick check on state obs - to ensure no flipping with large deltas. However, I've tried many, many different things, and unfortunately, any additional reward doesn't fix the problem - in fact, I believe the flipping was originally a local min solution, where hopping would result in greater reward, since the flipping doesn't always activate the standing reward portion of the hopping reward

currently testing out non delta joint pos

what do you mean non delta joint pos? all of them should use that controller

what do you mean non delta joint pos? all of them should use that controller
I tried pd_joint_pos instead of pd_joint_delta_pos is what I meant, didn't work though

I've diagnosed the problem:
Originally, I tried to implement the robot center of mass x velocity, but it didn't work, so I just used the xvelocity of the slider joint holding the hopper in the xz plane (see in current implementation of subtreelinvelx function).

Since the entire robot's center of mass x velocity isn't taken into consideration (and only the torso xvel which is attached to the slider), the robot isn't penalized for flipping in a large circle as long as its torso continues to move in the +x world frame direction

Here is original robot center of mass velocity function, which is slow and doesn't work

    @property  # dm_control mjc function adapted for maniskill
    def subtreelinvelx(self):
        """Returns linvel x component of robot"""
        total_mass = 0.0
        vels = []
        links = self.agent.robot.get_links()
        # skip first three dummy links
        for i in range(3,len(links)):
            vel = links[i].get_linear_velocity() # vel is (b,3)
            mass = links[i].get_mass()[0]
            vels.append(vel * mass.item())
            total_mass += mass.item()
        vels = torch.stack(vels, dim=0) # (#links, b, 3)
        com_vel = vels.sum(dim=0) / total_mass # (b, 3)
        return com_vel[:,0]

it seems like the get_linear_velocity() function doesn't return the center of mass linear velocity of the links?, which is causing the issue.

If calculating the center of mass velocity for an entire robot is non-trivial, this is an issue for hopper + all planar + all humanoid dm_control task implementations, since they com velocity as reward

Lemme get back to you. I think I also just found a bug with the GPU sim linear and angular velocities. CPU sim is correct but on GPU sim they might flipped. Working with @fbxiang now to debug it.

Actually you can use the torso link. It should work. That is the link they are reading the center of mass velocities from. All velocity data is center of mass actually.

e.g.

link = self.agent.robot.links_map["torso"]
print(link.linear_velocity, link.angular_velocity)

Also merge in main before doing more experiments. I just fixed #450 merged in #452 , so the link velocities used or com velocities are wrong lol

Thinking also more, maybe it is also better to just penalize the y-axis rotation, instead of velocity (its okay to go fast, just don't flip over).

**The results have large variance because in both hop and stand tasks, in seed 2, ppo learns to straighten its legs out too early, and doesn't explore tucking them in when hopper is on its belly, in order to get up - ppo then gets stuck at this local optimum in seed 2 - This is a a result of lack of exploration in ppo, not a fault of the environment, as shown below

Standing results and command:

for i in {1..3}; do python ppo.py --exp_name="standseed${i}" --env_id="MS-HopperStand-v1" --num_envs=4096 --update_epochs=8 --num_minibatches=32 --total_timesteps=30_000_000 --eval_freq=10 --num_eval_steps=600 --num_steps=100 --gamma=0.99 --ent_coef=0.01 --seed=${i}; done

Hopping results and command:

for i in {1..3}; do python ppo.py --exp_name="hopseed${i}" --env_id="MS-HopperHop-v1" --num_envs=4096 --update_epochs=8 --num_minibatches=32 --total_timesteps=60_000_000 --eval_freq=10 --num_eval_steps=600 --num_steps=100 --gamma=0.99 --ent_coef=0.01 --seed=${i}; done

RGB Hop results and command:

python ppo_rgb.py --exp_name="pixelhop" --env_id="MS-HopperHop-v1" --num_envs=512 --update_epochs=8 --num_minibatches=32 --total_timesteps=40_000_000 --eval_freq=15 --num_eval_steps=600 --num_steps=100 --gamma=0.99 --seed=1 --no-include-state

Local Optima in PPO training

seed 2 ppo local optimum:
ppo learns to straighten its legs out no matter what, and doesn't overcome this local optimum during training in seed 2

seed 1 and 3 don't have this issue:

seed 1 stand:

seed 3 stand:

**and even if the hopper learns to flip, it is a local minimum as it will eventually learn that hopping results in more rewards than flipping, this is because the center of mass of entire robot body reward correctly rewards hopping more than flipping

seed 3 of hopping command above first learns to flip, then later learns better how to hop to maximize the reward, eventually forgetting the flipping behavior:**
remnant of flipping behavior shown during middle of training: